Sketch-Based Estimation of Subpopulation-Weight

Summaries of massive data sets support approximate query processing over the original data. A basic aggregate over a set of records is the weight of subpopulations specified as a predicate over records’ attributes. Bottom-k sketches are a powerful summarization format of weighted items that includes priority sampling [18] (pri) and the classic weighted sampling without replacement (ws). They can be computed efficiently for many representations of the data including distributed databases and data streams. We derive novel unbiased estimators and efficient confidence bounds for subpopulation weight. Our estimators and bounds are tailored by distinguishing between applications (such as data streams) where the total weight of the sketched set can be computed by the summarization algorithm without a significant use of additional resources, and applications (such as sketches of network neighborhoods) where this is not the case. Our rank conditioning (RC) estimator, is applicable when the total weight is not provided. This estimator generalizes the known estimator for pri sketches [18] and its derivation is simpler. When the total weight is available we suggest another estimator, the subset conditioning (SC) estimator which is tighter. Our rigorous derivations, based on clever applications of the Horvitz-Thompson estimator (that is not directly applicable to bottom-k sketches), are complemented by efficient computational methods. Performance evaluation using a range of Pareto weight distributions demonstrate considerable benefits of the ws SC estimator on larger subpopulations (over all other estimators); of the ws RC estimator (over existing estimators for this basic sampling method); and of our confidence bounds (over all previous approaches). Overall, we significantly advance the state-of-the-art estimation of subpopulation weight queries.